resolve #42

Greptile Summary

This PR introduces a new HNSW-RaBitQ index type (HNSW_RABITQ) to zvec, combining the HNSW graph structure with RaBitQ scalar quantization for approximate nearest-neighbor search. The implementation spans a new hnsw_rabitq algorithm directory (~7000 lines), full builder/searcher/streamer support, Python bindings, DB-layer integration, and tests.

Key changes:

• New RabitqConverter (K-means clustering + rotation) and RabitqReformer (quantization + centroid look-up) components implement the two-phase offline/online quantization pipeline.
• HnswRabitqBuilder performs a two-phase build: first quantizes all vectors via RabitqReformer::convert, then builds the HNSW graph in parallel.
• HnswRabitqSearcher and HnswRabitqStreamer support all three search modes (standard, brute-force, and p-keys), including group-by and filter operations.
• DB schema validation for HNSW_RABITQ correctly gates on platform (Linux x86_64), CPU capabilities (AVX2/AVX512F), and data type (FP32).

Issues found:

• The stale error message for unsupported dense-vector index types (in schema.cc) lists only FLAT|HNSW|IVF and does not mention HNSW_RABITQ, which was added to the supported set in this PR.
• In hnsw_rabitq_query_algorithm.cc, after get_full_est is computed, nodes are unconditionally added to the candidates heap without re-checking the full estimate against the topk threshold, leading to unnecessary heap operations on the hot search path.
• Several issues identified in earlier review rounds (VLAs, null dereferences, POSIX-only headers, search_bf_by_p_keys_impl being a no-op, cluster->mount() return value ignored) have been addressed. The train(IndexTrainer::Pointer) overload leaving reformer_ null (and thus causing a crash in build()) remains unresolved per prior thread history.

Confidence Score: 3/5

• Not yet safe to merge — the train(IndexTrainer::Pointer) path leaves reformer_ null and will crash on any subsequent build() call; this should be resolved before merging.
• The core algorithm and search/stream paths are generally well-structured and most previously-flagged bugs have been fixed. However, the unresolved train(IndexTrainer::Pointer)→build() null dereference (tracked in prior review threads with no "fixed" reply) represents a crash-level regression on a documented public API path. The remaining open issues (unused <execinfo.h> include, stale error message, minor candidates-heap performance) are lower severity.
• Pay close attention to src/core/algorithm/hnsw_rabitq/hnsw_rabitq_builder.cc — specifically the train(const IndexTrainer::Pointer &) overload (lines 333–348) which does not initialize reformer_, and the subsequent build() call at line 401 that dereferences it.

Important Files Changed

Sequence Diagram

sequenceDiagram
    participant Caller
    participant HnswRabitqBuilder
    participant RabitqConverter
    participant RabitqReformer
    participant HnswRabitqEntity
    participant HnswRabitqAlgorithm

    Caller->>HnswRabitqBuilder: init(meta, params)
    HnswRabitqBuilder->>RabitqConverter: init(meta, params)
    HnswRabitqBuilder->>HnswRabitqEntity: init()
    HnswRabitqBuilder->>HnswRabitqAlgorithm: init()

    Caller->>HnswRabitqBuilder: train(holder)
    HnswRabitqBuilder->>RabitqConverter: train(holder) [K-means clustering]
    HnswRabitqBuilder->>RabitqConverter: dump(MemoryDumper)
    HnswRabitqBuilder->>RabitqReformer: init + load(MemoryReadStorage)
    Note over HnswRabitqBuilder: reformer_ now ready

    Caller->>HnswRabitqBuilder: build(threads, holder)
    loop For each vector
        HnswRabitqBuilder->>RabitqReformer: convert(vec) → quantized_vec
        HnswRabitqBuilder->>HnswRabitqEntity: add_vector(level, key, quantized_vec)
    end
    loop Parallel threads
        HnswRabitqBuilder->>HnswRabitqAlgorithm: add_node(id, level, ctx)
    end

    Caller->>HnswRabitqBuilder: dump(dumper)
    HnswRabitqBuilder->>RabitqConverter: dump(dumper) [centroids + rotator]
    HnswRabitqBuilder->>HnswRabitqEntity: dump(dumper) [graph data]

    Note over Caller: Search path
    Caller->>HnswRabitqSearcher: load(storage)
    HnswRabitqSearcher->>RabitqReformer: load(storage)
    HnswRabitqSearcher->>HnswRabitqEntity: load(storage)
    Caller->>HnswRabitqSearcher: search(query, count, ctx)
    HnswRabitqSearcher->>RabitqReformer: transform_to_entity(query) → QueryEntity
    HnswRabitqSearcher->>HnswRabitqQueryAlgorithm: search(QueryEntity, ctx)

_{Last reviewed commit: "Merge remote-trackin..."}